ci(perf): Track Firewood Performance via AvalancheGo Benchmarks #1493

Elvis339 · 2025-11-27T12:49:36Z

Why

Track C-Chain reexecution benchmark performance over time. Catch regressions before production.

Closes #1494

How

Firewood → triggers AvalancheGo benchmark → downloads results → publishes to GitHub Pages

Changes

scripts/bench-cchain-reexecution.sh
- Trigger AvalancheGo's C-Chain reexecution benchmark
- Poll for workflow registration
- Wait for completion and download artifacts
.github/workflows/track-performance.yml
- Orchestrate benchmark trigger via the script
- Publish results to GitHub Pages (main → bench/, branches → dev/bench/{branch}/)

Usage

# Auth
nix run ./ffi#gh -- auth login
export GH_TOKEN=$(nix run ./ffi#gh -- auth token)

# Predefined test
./scripts/bench-cchain-reexecution.sh trigger firewood-101-250k

# With specific Firewood version
FIREWOOD_REF=v0.0.18 ./scripts/bench-cchain-reexecution.sh trigger firewood-101-250k

# Custom block range
RUNNER=avalanche-avalanchego-runner-2ti \
FIREWOOD_REF=v0.0.18 \
CONFIG=firewood \
START_BLOCK=1 \
END_BLOCK=100 \
BLOCK_DIR_SRC=cchain-mainnet-blocks-200-ldb \
./scripts/bench-cchain-reexecution.sh trigger

# Other commands
./scripts/bench-cchain-reexecution.sh tests    # list available tests
./scripts/bench-cchain-reexecution.sh list     # list recent runs
./scripts/bench-cchain-reexecution.sh status <run_id>

Set FIREWOOD_REF=v0.0.18 explicitly. Without it, the workflow builds from HEAD, which currently fails due to changes in FFI layer

Added new benchmark script (bench-cchain-reexecution.sh) which
caused CI to fail with "Config does not cover the file". Shell
scripts aren't checked for license content (only .rs/.go/.h are),
but must be explicitly listed in the config. Exclude entire
scripts/ directory to avoid listing each script individually.

https://github.com/ava-labs/firewood/blob/main/.github/check-license-headers.yaml

rkuris

Early review, could use another pass

There's a lot of code here and I barely was able to complete the review in my maximum review time. Please consider breaking this up for more timely reviews, especially if the review is anything larger than this.

I'm also a little confused about how we track which firewood and avalanchego versions we ran the test against.

Let's say the performance loss was due to a change in avalanchego, how can we know that?

.github/workflows/track-performance.yml

rkuris · 2026-01-26T17:28:03Z

.github/workflows/track-performance.yml

+          GH_TOKEN: ${{ secrets.FIREWOOD_AVALANCHEGO_GITHUB_TOKEN }}
+          # Custom mode (ignored when test is specified)
+          CONFIG: ${{ inputs.config }}
+          START_BLOCK: ${{ inputs.start-block }}


It seems like if you set START_BLOCK you better also be setting CURRENT_STATE_DIR_SRC to let it know where to get the bootstrap database, is that correct?

If so, we should verify that either neither is provided or both are.

CURRENT_STATE_DIR is optional, not required because you might want to start from Genesis.

firewood/scripts/bench-cchain-reexecution.sh

Line 31 in cb0f6be

# CURRENT_STATE_DIR_SRC (optional) S3 state directory (empty = genesis run)

Validation:

firewood/scripts/bench-cchain-reexecution.sh

Line 219 in cb0f6be

[[ -z "${START_BLOCK:-}${END_BLOCK:-}${BLOCK_DIR_SRC:-}" ]] && \

If you want to start from genesis, then you either must set START_BLOCK to 0 (1?) or not set it. So, if START_BLOCK is not 0, then CURRENT_STATE_DIR_SRC must be set. Isn't that correct?

Close, START_BLOCK should be 1 (not 0) for Genesis, and when CURRENT_STATE_DIR_SRC is empty it means starting from genesis (no pre-existing state to bootstrap from). So the valid combinations are:

Genesis run: START_BLOCK=1, CURRENT_STATE_DIR_SRC empty

Resume run: START_BLOCK=N, CURRENT_STATE_DIR_SRC points to state at block N-1

Remove local developer tooling (justfile recipe, flake.nix, METRICS.md) to reduce PR scope. These will be submitted in a follow-up PR after the CI workflow changes are merged.

Elvis339 · 2026-01-26T18:24:07Z

Early review, could use another pass

There's a lot of code here and I barely was able to complete the review in my maximum review time. Please consider breaking this up for more timely reviews, especially if the review is anything larger than this.

I'm also a little confused about how we track which firewood and avalanchego versions we ran the test against.

Let's say the performance loss was due to a change in avalanchego, how can we know that?

I've split into two PRs per your feedback:

This PR (CI): Workflows, benchmark script, license config
#1642 (Local tooling): justfile recipe, METRICS.md docs, flake.nix

On version tracking github-action-benchmark stores time-series data (mgas/s over time) without version metadata tracing a regression requires digging through run history.

This is a conscious tradeoff to get data flowing now with minimal setup, then iterate to something more robust (S3 storage, richer metadata, export for analysis). Planning to revisit after ~2 weeks of data collection.

For now, each run logs Firewood / Avalanchego refs in the GitHub Actions summary, so the info exists just not queryable. Added your concern to the tracking doc as something to think through for the next iteration. How does that sound?

Summary example: https://github.com/ava-labs/firewood/actions/runs/21334552166

scripts/bench-cchain-reexecution.sh

Elvis339 · 2026-01-27T17:53:19Z

Executing:

RUNNER=avalanche-avalanchego-runner-2ti \
FIREWOOD_REF=v0.0.18 \
CONFIG=firewood \
START_BLOCK=1 \
END_BLOCK=100 \
BLOCK_DIR_SRC=cchain-mainnet-blocks-200-ldb \
./scripts/bench-cchain-reexecution.sh trigger

Triggered reexecution benchmark in AvalancheGo https://github.com/ava-labs/avalanchego/actions/runs/21408080471

scripts/bench-cchain-reexecution.sh

RodrigoVillar · 2026-01-27T20:42:48Z

scripts/bench-cchain-reexecution.sh

+err() {
+    if [[ "${GITHUB_ACTIONS:-}" == "true" ]]; then
+        echo "::error::$1"
+    else
+        echo "error: $1" >&2
+    fi
+}
+
+die() { err "$1"; exit 1; }


Could we combine these two into one (i.e. just append the exit code to err and remove die)?

Keeping them separate as err logs without exiting (useful if we need to log multiple errors before bailing), die is the "fatal" version. Clearer intent at call sites.

The only place we call err in bench_cchain_reexecution.sh is when we pass in an unknown command and even then, we exit afterwards with code 1.

Clearer intent at call sites.

I think replacing uses of die with err and exit 1 would be clearer in this case.

👍 Done

d5027de

scripts/bench-cchain-reexecution.sh

.github/workflows/track-performance.yml

rkuris · 2026-01-27T21:56:25Z

.github/workflows/track-performance.yml

+          GH_TOKEN: ${{ secrets.FIREWOOD_AVALANCHEGO_GITHUB_TOKEN }}
+          # Custom mode (ignored when test is specified)
+          CONFIG: ${{ inputs.config }}
+          START_BLOCK: ${{ inputs.start-block }}


If you want to start from genesis, then you either must set START_BLOCK to 0 (1?) or not set it. So, if START_BLOCK is not 0, then CURRENT_STATE_DIR_SRC must be set. Isn't that correct?

.github/workflows/track-performance.yml

…ith err and intentional exit code status

Elvis339 added 3 commits November 27, 2025 16:39

ci: track performance

1ea99bb

test(ci): add PR label trigger for testing

1ce1481

Elvis339 added the run-benchmark label Nov 27, 2025

temp fix ci

8794250

Elvis339 removed the run-benchmark label Nov 27, 2025

Elvis339 changed the title ~~Es/enable firewood dev workflow~~ ci: Track Firewood Performance via AvalancheGo Benchmarks Nov 27, 2025

Elvis339 changed the title ~~ci: Track Firewood Performance via AvalancheGo Benchmarks~~ ci(perf): Track Firewood Performance via AvalancheGo Benchmarks Nov 27, 2025

Elvis339 marked this pull request as ready for review November 27, 2025 15:04

Elvis339 requested review from aaronbuchwald, demosdemon and rkuris as code owners November 27, 2025 15:04

Elvis339 mentioned this pull request Nov 27, 2025

Track Firewood Performance via AvalancheGo Reexecution Benchmarks #1494

Open

Elvis339 marked this pull request as draft November 27, 2025 15:11

Elvis339 self-assigned this Dec 1, 2025

Elvis339 added 4 commits December 2, 2025 22:13

ci: use switch CI token

4aa0e08

ci: lint

32fd06f

fix

7a75021

docs

e35fa5d

Elvis339 marked this pull request as ready for review December 2, 2025 20:32

Merge branch 'main' into es/enable-firewood-dev-workflow

83a113a

RodrigoVillar reviewed Dec 2, 2025

View reviewed changes

README.md Show resolved Hide resolved

RodrigoVillar reviewed Dec 4, 2025

View reviewed changes

.github/workflows/track-performance.yml Show resolved Hide resolved

Elvis339 added 3 commits December 4, 2025 20:54

ci: push performance to benchmark data

18f4035

ci(perf): add benchmark workflow with nix-based just commands

73fc781

Merge branch 'es/enable-firewood-dev-workflow' of https://github.com/…

a19f1e8

…ava-labs/firewood into es/enable-firewood-dev-workflow

Elvis339 requested a review from alarso16 as a code owner December 4, 2025 17:17

docs

668cef3

RodrigoVillar requested changes Dec 4, 2025

View reviewed changes

METRICS.md Outdated Show resolved Hide resolved

Elvis339 added 2 commits January 25, 2026 16:07

ci(gh-pages): temp. add workflow_dispatch to rebuild Pages

7d12619

fix(gh-pages)

09e9f46

Elvis339 had a problem deploying to github-pages January 25, 2026 15:18 — with GitHub Actions Failure

ci(gh-pages): remove temp. set workflow_dispatch

c7cad22

Elvis339 commented Jan 25, 2026

View reviewed changes

refactor(bench-cchain-reexecution): rename verify_run_inputs to run_i…

84b381a

…nputs_match reducing cognitive load

Elvis339 requested a review from rkuris January 26, 2026 14:58

Elvis339 marked this pull request as ready for review January 26, 2026 16:39

Elvis339 and others added 3 commits January 26, 2026 20:39

Merge branch 'main' into es/enable-firewood-dev-workflow

a8db04d

chore: exclude all scripts from license header check

18d4047

Merge branch 'es/enable-firewood-dev-workflow' of https://github.com/…

79a2def

…ava-labs/firewood into es/enable-firewood-dev-workflow

Elvis339 commented Jan 26, 2026

View reviewed changes

rkuris requested changes Jan 26, 2026

View reviewed changes

Elvis339 mentioned this pull request Jan 26, 2026

chore(track-performance): local iteration #1642

Open

chore: split PR - extract local tooling to separate PR

cb0f6be

Remove local developer tooling (justfile recipe, flake.nix, METRICS.md) to reduce PR scope. These will be submitted in a follow-up PR after the CI workflow changes are merged.

RodrigoVillar reviewed Jan 26, 2026

View reviewed changes

scripts/bench-cchain-reexecution.sh Show resolved Hide resolved

Elvis339 requested review from RodrigoVillar and rkuris January 26, 2026 18:51

Elvis339 commented Jan 26, 2026

View reviewed changes

scripts/bench-cchain-reexecution.sh Show resolved Hide resolved

Elvis339 removed the DO NOT MERGE This PR is not meant to be merged in its current state label Jan 27, 2026

chore(bench-cchain-reexecution): replace hardcoded AvalancheGo branch

5ef9313

RodrigoVillar requested changes Jan 27, 2026

View reviewed changes

rkuris approved these changes Jan 27, 2026

View reviewed changes

Elvis339 and others added 2 commits January 28, 2026 11:38

docs: clarify benchmark workflow comments and simplify env var docs

809c8ea

Merge branch 'main' into es/enable-firewood-dev-workflow

2236393

Elvis339 requested a review from RodrigoVillar January 28, 2026 10:40

Elvis339 mentioned this pull request Jan 28, 2026

ci: add scheduled benchmark runs (#1639) #1645

Open

3 tasks

chore(bench-cchain-reexecution): remove die function and repalce it w…

d5027de

…ith err and intentional exit code status

+                    # Structure on benchmark-data branch (see track-performance.yml for how this is populated):
+                    #   bench/              - Official benchmark history (main branch only)
+                    #   dev/bench/{branch}/ - Feature branch benchmarks (experimental)
+                    - name: Include benchmark data

                     "**/tests/compile_*/**",
                     "justfile",
-                    "scripts/run-just.sh",
+                    "scripts/**",

ci(perf): Track Firewood Performance via AvalancheGo Benchmarks #1493

Are you sure you want to change the base?

ci(perf): Track Firewood Performance via AvalancheGo Benchmarks #1493

Conversation

Elvis339 commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

How

Changes

Usage

Related

Uh oh!

Elvis339 commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rkuris left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Elvis339 commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Elvis339 commented Jan 27, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Elvis339 commented Nov 27, 2025 •

edited

Loading

Elvis339 commented Nov 27, 2025 •

edited

Loading

rkuris left a comment •

edited

Loading

Elvis339 commented Jan 26, 2026 •

edited

Loading